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Executive 



Debates over the use of college rankings are often framed in binaries: ardent advocates versus outright 
rejectionists, rankers versus rankees. But the American higher education landscape is complex, and 
so too should be our discussions of rankings. Moreover, the policymaking process is simultaneously 
intricate and predictable — information is processed and evaluated, subjected to the judgments of 
policymakers, shaped to support hardened policy positions, and ultimately used to justify the outcomes 
of an inherently political process. College rankings are part of the information policymakers may 
consider as they usher higher education policy through this complex process. Yet limited information 
exists on the role college rankings might play in policymaking for postsecondary education, and few 
suggestions have been offered for how policymakers could use college rankings to support and 
advance public policy goals. 



This brief provides an overview of nationai postsecondary 
assessment efforts and notes the simiiarities and differences 
these approaches have taken in comparison to coiiege rankings, 
presents severai exampies of the inciusion of coiiege rankings 
in state government assessment efforts, highiights key find- 
ings, and makes recommendations for enhancing the poiicy 
reievance of coiiege rankings. Key findings and recommenda- 
tions are drawn from a review of the iiterature on coiiege rank- 
ings and postsecondary government accountabiiity systems, 
and from interviews with individuais from federai and state 
government agencies, trade associations, and other groups. 



Key Findings 

A durabie fixture in the marketpiace of information on coiieges, 
rankings inform pubiic notions of coiiege quaiity. 

Coiiege rankings are one way stakehoiders of higher education 
obtain information on institutions and construct notions of 
educationai quaiity. The rise in popuiarity of rankings and their 
durabiiity are the resuit of severai conditions, inciuding 
increased student mobiiity, institutionai use of rankings in 
promotionai campaigns and decision-making, and the use of 
rankings in internationai discussions of assessment, account- 
abiiity, and quaiity assurance efforts. 
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Data limitations restrict the usefulness of college rankings 
to policymakers. 

The usefulness of college rankings is limited by the availability 
of credible and comparable data indicators. In some instances, 
particularly at the state level, policymakers may find that data 
compiled by their own jurisdiction through student unit record 
systems paint a more comprehensive picture of the successes 
and challenges of higher education than data used in the 
construction of ranking schemes. 

The structure of rankings limits the transference of information 
relevant to policy. 

Ordinal rankings and benchmarking dilute the relevance of infor- 
mation policymakers derive from college rankings. Presenting infor- 
mation in this way might be appropriate to assess how institutions 
are doing comparatively, but it is of limited use in crafting answers 
to questions important to policymakers such as “Are we producing 
the educational outcomes we desire?”, “Are students learning?”, 
and “What improvements have resulted from policy changes?” 

Rankings have the potential to shift institutional behaviors in 
ways that may negatively affect policy goals. 

Rankings create incentives for institutions to take actions 
designed to improve their positions. This reactivity creates 
conditions in which institutions respond to the concept of 
educational quality embedded in rankings, which is not always 
aligned with public policy goals, particularly in the areas of 
equity and diversity. 



Recommendations 

Prior to using college rankings in the construction of public 
policy, policymakers need to consider the effects rankings 
can have on institutional behaviors, definitions of educational 
quality, and postsecondary equity outcomes. In support of 
these goals, the following recommendations are offered: 

Take precautions to ensure that college rankings are used as 
only as part of overall system assessment efforts and not as a 
stand alone evaluation of colleges. 

If policymakers are to use college rankings as part of assess- 
ment systems, they must be careful to do so as part of compre- 
hensive efforts designed to improve institutional performance 
toward public policy goals. Although rankings can serve as one 
metric for assessing institutional progress, policymakers should 
only use them as part of a larger feedback loop that leads to 
desired changes in institutional actions and policy. Using rank- 
ings as stand alone measures of institutional success seldom 
serves public policy goals. 

Support the collection of data that can be used to craft more 
policy-relevant college rankings, including providing funds to 
higher education institutions to widely implement and publish 
the results of student learning assessments. 

To be more relevant to policymakers, college rankings need to 
rely on indicators that reflect policy priorities and capture the 
skills that make students competitive and productive in a 
modern technological economy. Focusing on student learning 
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outcomes is one way rankings could become more relevant to 
policymakers, and policy efforts should support the develop- 
ment and implementation of instruments to collect meaningful 
learning outcome data. 

Leverage public attention to college rankings to shape general 
notions of college quality and advance equity goals. 
Policymakers should leverage the attention paid to college 
rankings to shape public discussions of educational quality to 
advance equity goals, an Increasingly important undertaking 
given shifting demographic patterns and the need to better 
serve students of color, adult learners, and low-income popula- 
tions to reach national educational attainment goals. ^ 
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Introduction 



In the past decade, a consensus has emerged on a renewed public agenda for higher education in 
the United States. Diverse groups encompassing America’s social landscape — including government 
actors, business interests, community organizations, foundations, and educational institutions — have 
articulated a common purpose for higher education: to increase quality, affordability, and production 
in the higher education pipeline, and to do so without significant new injections of public resources. 
To achieve these goals, the collection and dissemination of information on colleges^ is important to 
benchmark progress toward policy goals, assess college quality, and inform stakeholders about the 
complexity of the higher education enterprise. 



College rankings are one form of information about higher 
education. Rankings transmit simplified information on colleges 
to consumers, stimulate competition between institutions, and 
influence institutional policy. When they are designed with a 
clear purpose, constructed on reliable data, and developed 
with transparent and appropriate methodologies, college rank- 
ings hold the promise of Increasing salience of college quality 
to wide and diverse audiences. 

The use of college rankings, however, has not been without 
controversy as stakeholders of higher education have clashed 
over definitions of educational quality, how rankings can best 
capture quality, and the role rankings should play in shaping 
policy and institutional practice. College rankings have received 
significant attention in the popular press and academic jour- 
nals, in policymaking circles, and in the higher education 
community. The discourse has typically focused on the strengths 
and weaknesses of rankings, possible revisions to the methods 



employed to construct rankings, and the use of rankings in the 
areas such as institutional promotion and college choice. 

Consideration of how rankings might influence and interact 
with government policymaking efforts is notably absent from 
the discourse. Limited information exists on the role college 
rankings might play in postsecondary education policymaking, 
and few suggestions have been made for how policymakers 
might use rankings to support and advance public policy goals. 
This lack of dialogue may be a result of the quandary college 
rankings present to policymakers. On one hand, rankings are 
popular with the public, provide a means of sorting colleges on 
a national and increasingly international scale, and support an 
expansion of uniform data collection and reporting strategies — 
all factors that hold currency in postsecondary education policy 



^ For the sake of brevity, the term “college" is generally used to refer to institutions of postsec- 
ondary education. 
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environment. On the other hand, existing college rankings 
cover only a limited portion of the higher education landscape, 
have the potential to incentivize behaviors that run counter to 
public policy goals, and are based on educational inputs at a 
time when educational outcomes are of increasing importance 
to policymakers. 

This brief builds on previous and ongoing research and 
programming undertaken by the Institute of Higher Education 
Policy (IHEP) in the areas of accountability and college rankings.^ 
The brief begins with a review the broad landscape of college 
rankings and postsecondary assessment systems. Following 
sections present examples of the inclusion of college rankings 



in state government assessment efforts, highlight key findings, 
and make recommendations for enhancing the policy rele- 
vance of college rankings. Key findings and recommendations 
are drawn from a review of the literature on college rankings 
and postsecondary government accountability systems and 
from interviews with individuals from federal and state govern- 
ment agencies, trade associations, and other groups. ^ 



* For example, see “Making Accountability Work: Community Colleges and Statewide Higher 
Education Accountability Systems” (2006) and “Impact of College Rankings on Institutional Deci- 
sion Making: Four-Country Case Study” (2009). Additional resources on rankings are available 
from IHEP’s Ranking System Clearinghouse at www.ihep.org/Research/rankingsystemsclearing- 
house.cfm. 
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Policymaking, Postsecondary 
Assessment, and 

Rankings 




Over the past several decades, government entities, trade associations, and postsecondary institutions 
have established and implemented numerous measurement and assessment systems to capture 
information on inputs and outputs in the higher education sector. These efforts have generated vast 
amounts of data that construct and reflect notions of educational quality, and inform the public and 
policymakers on various aspects of postsecondary education. 



The continued demand for data on the higher education sector 
reflects a larger movement in U.S. politicai institutions toward 
evidenced-based policymaking. In this type of policymaking, 
decisions are informed by analysis using sound and transparent 
data to help identify issues, inform policy choices and program 
design, monitor policy implementation, and forecast future 
conditions and public needs (Scott 2005; Urban Institute 2008). 

College rankings and postsecondary assessment/account- 
ability efforts are two areas in which social measurements of 
higher education are used to communicate educational quality 
to broad audiences. The results of these ranking systems and 
assessment efforts hold particular currency for policymakers 
who seek data-driven information on which to base their deci- 
sions. Although this brief focuses on college rankings, a review 
of the national postsecondary assessment landscape is instruc- 
tive. These assessment efforts specifically avoid explicit rank- 
ings of colleges; instead, they focus on issues of access, cost, 
affordability, quality of the student experience, and transpar- 
ency, with aims to provide loosely comparable information to 
varying stakeholders (State Higher Education Executive Offi- 
cers 2005; U.S. Department of Education 2006). Several recent 
and ongoing efforts have shaped conversations on assess- 
ment and accountability at the national level. 



U.S. Secretary of Education’s Commission 
on the Future of Higher Education 

As significant investors in higher education, federal policy- 
makers maintain considerable interest in ensuring that colleges 
are accountable for public funds. Quality, transparency, and 
program assessment have developed as key elements of a 
public accountability agenda in higher education. 

In 2005-06, the U.S. Department of Education (ED) led a 
national dialogue on assessment and accountability by estab- 
lishing the U.S. Secretary of Education’s Commission on the 
Future of Higher Education (Spellings Commission). The forma- 
tion of the bipartisan commission demonstrated renewed 
federal interest in the quality of the nation’s colleges. The 
commission, which was composed of leaders in academia, 
business, and philanthropy, was charged with examining the 
accessibility, affordability, accountability, and quality of American 
higher education (ED 2006). The commission — at times contro- 
versial — released a final report outlining key findings and recom- 
mendations intended to make American higher education “more 
nimble, more efficient, and more effective” (ED 2006, p. xiii). 

A central point of the report was a concern about the lack of 
comparable college-to-college information, particularly in the 
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area of student learning outcomes. While it avoided the topic of 
ranking institutions, the commission did note that “parents and 
students have no solid evidence, comparable across Institu- 
tions, of how much students learn in colleges or whether they 
learn more at one college than another” (ED 2006, p. 14). To 
address this perceived information void, the commission made 
a number of recommendations regarding assessment of insti- 
tutional activities and student learning outcomes. To support 
the development of assessment metrics, the ED awarded a 
$2.45 million grant to a consortium of three higher education 
associations through the Fund for the Improvement of Postsec- 
ondary Education (ED 2007).^ The consortium was to develop 
valid and reliable measures to assess student achievement in 
higher education to inform students and families, policymakers, 
and institutions about colleges’ ability to produce an educated 
citizenry prepared to compete in the global marketplace (ED 2007). 

The Spellings Commission’s focus on assessment of institu- 
tional effectiveness pushed the issue onto the broader public 
policy stage and created an expectation that colleges should 
provide evidence of institutional effectiveness in ways that 
could be quantitatively measured and clearly reported. 
However, there was no explicit intent in the commission’s work 
to generate a competitive ranking of colleges on any one or set 
of measurements. 

In response to the environment created by the commission’s 
work, and motivated in part by the possibility of a mandated 
federal regulatory framework, different sectors of the higher 



® The consortium consisted of the Association of American Colieges and Universities (AACU), the 
Nationai Association of State Universities and Land-Grant Colleges (NASULGC), and the American 
Association of State Coileges and Universities (AASCU). 



education community have designed distinct approaches for 
disseminating information to stakeholders of postsecondary educa- 
tion — approaches that have similarly avoided ranking colleges. 

Voluntary System of Accountability Program 

The Voluntary System of Accountability (VSA) program is a 
collaborative effort involving four-year public postsecondary 
institutions represented by the American Association of State 
Colleges and Universities (AASCU), the Association of Public 
and Land-Grant Universities (APLU), and the public higher 
education community. Conceptualized in 2006, the VSA is 
designed to improve understanding of how public four-year 
colleges and universities operate through the compilation of 
loosely comparable information on each participating institu- 
tion (Shulenburger, Mehaffy, & Keller 2008). Institutions must 
elect to be part of the VSA project. 

For participating institutions, data are presented in the form of 
standardized five-page Web-based College Portraits.'' Each 
portrait contains basic information in three general categories: 
consumer information, student experiences and perceptions of 
the college experience, and student learning outcomes. Institu- 
tions are required to provide data in all categories, but they 
have a four-year lag period from the start date of their participa- 
tion to meet this requirement. The VSA is an effort by a large 
sector of the higher education community to report information 
on three critical areas in a comparable form (Shulenburger et 
al. 2008). By design, the project does not rank institutions on 
any individual or set of indicators. 



'' To see these portraits, go to www.collegeportraits.org. 
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University and Coiiege Accountabiiity Network 

Similar in intent to the VSA, the University and College Account- 
ability Network (U-CAN) is an initiative to provide higher educa- 
tion stakeholders with information on private four-year colleges 
and universities. Sponsored by the National Association of Inde- 
pendent Colleges and Universities — whose members include 
the nation’s nearly 800 private colleges — U-CAN was launched 
in 2007 in response to federal and state government calls for 
greater transparency in information (U-CAN 2009). 

U-CAN provides a Web-based interface through which users 
can search profiles of institutions from a selection of data points®. 
The profiles include information about admissions, academics, 
student demographics, graduation rates, common fields of 
study, institutional accreditation, faculty information, class size, 
tuition and fee trends, price of attendance, financial aid, campus 
housing, student life, and campus safety. In addition to this 
basic information, U-CAN profiles report institutional data on 
average total loan amounts at graduation, undergraduate class- 
size breakdowns, and net tuition price. U-CAN aspires to be 
holistic in its representation of institutions, providing key statis- 
tical data complemented by narrative descriptions and subject- 
specific links to relevant institutional Web pages (U-CAN 2009). 

U-CAN and the VSA differ in one important aspect — the latter 
reports on student learning outcomes and the student experi- 
ence. Institutions that participate in the VSA are required to report 
student learning outcome information on one of three standard- 
ized examinations: the Measure of Academic Proficiency and 
Progress (MAPP); the Collegiate Learning Assessment (CLA); or 
the Collegiate Assessment of Academic Proficiency (CAAP). 



® To see these portraits, go to www.ucan-network.org. 



Data on student experiences are reported from one of four 
surveys on student engagement, including the widely used 
National Survey of Student Engagement (NSSE). U-CAN does 
not require similar information to be reported in its institutional 
Web profiles. Despite the VSA making efforts to report student 
learning outcomes, the usefulness of the data is somewhat 
limited. Institutions may report outcomes using any one of 
several measuring instruments, so comparability across institu- 
tions is difficult. Debate is ongoing regarding the appropriateness 
of comparing college-to-college outcomes of student learning 
and engagement (Kuh 2007), but the lack of widespread unifor- 
mity in measuring instruments to-date makes this a moot point. 

The VSA and U-CAN initiatives exemplify the response of the 
higher education sector to calls from stakeholders for clear and 
comparable information on the college experience and student 
learning. The two programs do not report information on institu- 
tions in ways that would permit rankings to be generated on any 
single measurement point or group of points, but rather focusing 
on providing transparent information on individual institutions. 

Measuring Up — National Report Card on Higher Education 

Taking a different tack, the Measuring Up series has focused on 
the U.S. state as the unit of analysis, generating direct grading 
metrics to facilitate comparisons. Since 2000, the National 
Center for Public Policy and Higher Education has prepared 
report cards on postsecondary education performance in each 
of the 50 states and for the nation as a whole. These report 
cards, published in the biannual Measuring Up report, grade 
each state and the nation on performance indicators that eval- 
uate the states’ ability to provide educational opportunities 
beyond high school through the bachelor’s degree. The intent 
of the report cards is to provide the general public with informa- 
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tion on state-by-state performance and to challenge states to 
improve their performance (National Center for Public Policy 
and Higher Education 2008). 

Performance is measured across six categories: preparation, 
participation, affordability, completion, benefits, and learning. 
Grades result from a multistep procedure that includes selecting 
indicators for each category, weighting each indicator according 
to its importance, identifying the third-highest performing state 
for benchmarking purposes, converting raw scores to indexes 
on the basis of the benchmark, and assigning letter grades 
based on accumulated index scores. This process produces 
grades for each state in each category and shows performance 
relative to other states.® Grades for “learning” have not been 
issued because of the lack of uniform and comparable indica- 
tors. According to the authors of Measuring Up, “All states 
receive an ‘incomplete’ in Learning because there are not suffi- 
cient data to allow meaningful state-by-state comparisons” 
(National Center for Public Policy and Higher Education 2008). 
Each state’s report card offers a detailed synopsis of strengths 
and weaknesses, summary information, and grade changes 
over time. 

Although not an overt ranking, the Measuring Up series does 
evaluate states on a set of common indicators in a way that is 
easily understandable by end-users. It may not be explicitly 
clear what a “B” in affordability means or what the experiential 
difference between a “B” and a “C” would be at a postsec- 
ondary institution; but people understand that a “B” is somehow 
better than a “C” — a simple understanding that is lacking in the 



more nuanced state or institutional information presented in 
other assessments. Moreover, the comparability of grades 
across states fits into the narrative of competitiveness that 
underlies much of the political debate around postsecondary 
policy issues; particularly the desire to be recognized by 
constituents’ and economic interest alike as providing quality 
postsecondary education. 

Reviewing the national higher education assessment landscape 
illustrates a trend toward assessing college quality and facili- 
tating general comparisons across similar institutions. Federal 
and state governments, the postsecondary education commu- 
nity, and the non-profit sector have each advanced various 
systems to measure institutional performance and convey infor- 
mation on higher education to the public. However, there is an 
absence of government or postsecondary sector systems that 
directly rank institutions. Filling this void, third-party college 
rankings have become viable instruments for sorting postsec- 
ondary institutions and providing some level of information 
about college quality to policymakers and the wider public. ^ 



® A description of this method of computing state grades is avaiiabie at http://measuringup2008. 
highereducation.org/print/technicai_guide.pdf. 
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State Assessment 
Approaches and 
College Rankings 



Rankings offer one way to reduce the asymmetry of information between higher education institu- 
tions and sector stakeholders. The popularity of rankings suggests that despite the proliferation of 
information about colleges provided by government assessment efforts, institutions themselves, and 
various third parties, stakeholders gravitate toward systems that provide some interpretation of the 
information. Although they are often criticized for arbitrary weighting and unsubstantiated definitions 
of “quality,” rankings provide the public at least some information in an easily digestible format. 



Because notions of educational quality depend on many inputs 
and outputs, it is a difficult concept to capture. Nonetheless, 
general demands for social measurements and specific calls for 
hard Information on the quality of higher education have led to the 
development of college rankings by a variety of public and private 
sources in a national and increasingly international context. 
Appendix A describes several ranking systems available in the 
marketplace, highlighting their methodologies and data sources. 

Determining whether and how policymakers, particularly at the 
state level, use college rankings in the decision-making process 
is difficult. Because of the simplicity with which rankings 
transmit notions of educational quality, there would seem to be 
a great temptation for policymakers to use rankings as a proxy 
to see how one jurisdiction’s higher education sector compares 
to others. And because governments act in a competitive 
marketplace to attract high-quality students and educationally 
driven economic growth, incentives exist for policymakers to 
seek to have postsecondary education in their state be recog- 
nized as “the best.” However, although it is conceivable that 
rankings would appeal to state policymakers, the interviews 



and analysis undertaken for this brief revealed little in the way 
of direct use of rankings in policy formation, although it was 
revealed that policymakers are being exposed to college rank- 
ings in various ways. The following section draws on the expe- 
riences of three states to illustrate how college rankings have 
been folded into comprehensive government assessment 
efforts, increasing the salience of college rankings with policy- 
makers and suggesting at least a marginal impact of rankings 
in the policy process. 

Minnesota 

Minnesota has a well-developed, comprehensive set of post- 
secondary education goals designed to support economic and 
civic development (Minnesota Office of Higher Education 
2009). Goals for postsecondary education in the state are 
outlined in the annual Minnesota Measures report, compiled by 
the Minnesota Office of Higher Education and delivered to the 
legislature, governor, and general public. The report is intended 
to help Minnesota policymakers assess the goals and effective- 
ness of higher education in the state (Minnesota Office of 
Higher Education 2009). 
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TABLE 1 



Minnesota Measures Presentation of Rankings 



Newsweek ranked the University of 
Minnesota 30th internationaiiy among 
research institutions in 2007. 



China’s institute of Higher Education at 
Shanghai Jiao Tong University ranked the 
University of Minnesota 28th internationaiiy 
among the top 100 research institutions in 
2008. This was an improvement from 33rd 
in 2007 and rankings in the iow 30s in 
previous reports. 



The London Times Higher Education 
Suppiement ranked the University of 
Minnesota 87th in 2008. This was an 
improvement from 142 in 2007 and 187 
in 2006. The ranking methodoiogy for 
this report is broader than that used 
in other studies. 



SOURCE: WWW.OHE.STATE.MN.US/PDF/MINNESOTAMEASURES2009.PDF 



Minnesota Measures uses various indicators to report on the 
functioning of the state’s public postsecondary institutions. 
Among them is an indicator designed to assess the University of 
Minnesota governing board’s goal to be “among the top three 
public research universities in the world” (Minnesota Office of 
Higher Education 2009, p. 54). Progress toward this goal is 
measured primarily by the institution’s position on several 
ranking metrics, with an emphasis on the Center for Measuring 
University Performance’s “Top American Research Universities” 
rankings. The report outlines the criteria used to rank the institu- 
tions and discusses the placement of the University of Minne- 
sota in the overall ranking. Noting the domestic tilt of the “Top 
American Research Universities” rankings, the report also 
includes the University of Minnesota’s ranking on several inter- 



national ranking systems, such as those compiled by the 
Shanghai Jiao Tong University, the London Times Higher Education 
Suppiement, Newsweek, the G-factor International University 
Rankings, and Webometrics. TABLE 1 provides an example of 
how ranking information is reported in Minnesota Measures. 

Contextual information is lacking on how the rankings are 
developed, the audience for which each ranking is intended, 
and the methodology used to create the ranking. Readers of 
the Minnesota report must determine for themselves the rele- 
vance of these measurements to state policy goals beyond the 
objective of having a globally recognized public research 
university. Including rankings in the report allows state policy- 
makers to see how state institutions rank in comparison with 
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similar institutions. This information may be of limited policy 
relevance, however, due to a lack of discussion on how a 
ranking placement impacts state policy goals. 

Indiana 

In 2007, Indiana’s Commission for Higher Education (ICHE) artic- 
ulated specific initiatives and recommendations to support state 
development goals (ICHE 2007). Goals were established in six 
key areas: college completion, affordability, preparation, commu- 
nity college, major research universities, and accountability. In 
establishing goals for major research universities in the public 
system, the ICHE encouraged institutions to develop strategies 
to move toward becoming top national research universities. 

The state identifies college ranking systems as one method of 
validating institutional progress toward the policy goal of devel- 
oping top research universities (ICHE 2008). Ranking systems 
mentioned in the ICHE’s report include “Top American Univer- 
sities” from the Center for Measuring University Performance; 
the “Academic Ranking of World Universities” produced by 
Shanghai Jiao Tong University; and “America’s Best Colleges,” 
produced by U.S. News & World Report (USNWR). These ranking 
systems were selected because of their popularity with the 
general public and or the academic community, the consistent 
format in which they are published, and the wide range of 
metrics used in compiling them (ICHE 2008). 

Rankings are presented in graphical format for Purdue Univer- 
sity, Indiana University, and identified peer institutions, with 
limited contextual information on the methods used to create 
the rankings. The ICHE explicitly states that it intends to monitor 



the rankings with the goal of “moving our major research 
universities to higher levels in the rankings” (ICHE 2008, p. 6). 
This suggests that the metrics used to develop the rankings are 
important for both institutional leaders and state decision 
makers to consider in the development of institutional and state 
policy. As with Minnesota, the lack of contextual discussion on 
methodology and the purpose of the rankings limits the policy 
relevance of the information presented in the report. 



Texas 

Texas provides another example of how college rankings are 
being folded into overall state assessment and accountability 
efforts. The University of Texas (UT) System Board of Regents 
strategic plan outlines clear expectations — built around six key 
indicators — for each of the 15 institutions in the state system 
(UT System 2006). 

The strategic plan calls for actions and public investments to 
increase the quality of the system’s institutions. One measure- 
ment of quality improvements is college rankings, which are 
used to track the “progress and impact of these investments — 
to recruit top faculty and build state-of-the-art research facilities, 
to enhance technology transfer, to attract and retain a diverse 
group of students — even though the rankings themselves 
should not be a strategic goal” (UT System 2008, sec. II.5.17). 

Positions in college rankings are included in both individual 
institution accountability profiles and the overall system report. 
Rankings are presented in three broad areas — research 
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TABLE 2 



Shanghai Jiao Tong University Academic Ranking of Worid Universities 200712008 



INSTITUTION AND PROGRAMS RANK INSTITUTION RANK AMONG 

AMONG 500 WORLD UNIVERSITIES 166 AMERICAN UNIVERSITIES 



UT Austin 


38 


29 


Engineering/Technology/Computer Sciences 


6 




Naturai Sciences and Mathematics 


29 




Social Sciences 


19 




UT Southwestern Medicai Center 


39 


30 


Ciinicai Medicine and Pharmacy 


7 




Life and Agricuiturai Sciences 


6 




UT HSC-Houston 


151-202 group 


71-88 group 


Ciinicai Medicine and Pharmacy 


31 




UT M.D. Anderson Cancer Center 


151-202 group 


71-88 group 


Ciinicai Medicine and Pharmacy 


21 




UTHSC-San Antonio 


203-304 group 


89-117 group 


Ciinicai Medicine and Pharmacy 


52-75 group 




UT Medicai Branch 


203-304 group 


89-117 group 


Ciinicai Medicine and Pharmacy 


52-75 group 




UT Daiias 


305-402 group 


118-140 group 



SOURCE: WWW.UTSYSTEM.EDU/OSM/ACCOUNTABILITY/2007/ACCOUNTABILITYREPORT07-08.PDF, SECTION II. 5. 20 



productivity, technology transfer, and students — and include 
brief discussions of how they are compiled and how they 
should be interpreted (UT System 2008). For example, in 
discussing USNWR’s “America’s Best Colleges” rankings, the 
accountability profiles note that the top schools seldom change 
and that sustaining a ranking from year to year requires contin- 
uous improvement. 

The following ranking systems are cited in the UT strategic plan: 

• Research productivity: Shanghai Jiao Tong University; 
National Science Foundation; and the Center for Measuring 
University Performance. 

• Technology transfer: The Scientist magazine; U.S. Patent and 
Trade Office; Milken Institute (biotech patents). 

• Student-focused rankings: U.S. News and Worid Report’s 
“America’s Best Colleges” and “America’s Best Graduate 
Schools”; Diverse Issues in Higher Education. 

TABLE 2 illustrates how outcomes of one international ranking 
are reported in the overall UT System accountability report. 

The inclusion of college rankings into broader assessment 
efforts in Texas is instructive for several reasons. First, rankings 
are not used as stand-alone metrics of success or failure but 
are presented along with broad policy goals and with contextu- 
alizing information. By presenting rankings alongside policy 
goals (e.g., retaining a diverse group of students), the account- 
ability report shows why the ranking is included and what it is 



intended to convey to policymakers and other constituents. 
Second, the accountability reports include a number of rank- 
ings that capture different elements of the system and present 
information relevant to specific policy goals. For example, rank- 
ings on diversity outcomes are included to demonstrate the 
system’s achievements or challenges in this area. Including 
several ranking measures reduces the opportunity to game 
any one ranking for the purpose of state accountability efforts. 
Third, the rankings include detailed explanations of methods 
and data used in their computation. This information is impor- 
tant to inform policymakers and others on how rankings are 
developed and allow them to decide whether the included vari- 
ables are relevant to policy priorities. Finally, trending informa- 
tion on an institution’s position in the rankings is included to 
ensure that accountability reports are not just a single-year 
snapshot of performance. 

The use of college rankings in assessing higher education in 
Minnesota, Indiana, and Texas provides state policymakers 
access to ranking information within a larger assessment 
framework. The breadth of rankings presented to policymakers, 
the descriptions of how rankings are tabulated, and efforts to 
tie the rankings information to state policy goals demonstrate a 
partial attempt to use college rankings to inform policy devel- 
opment. The direct impact of rankings on policy evaluation and 
development is difficult to track. However, the presence of college 
rankings in state assessment reports indicates that they are more 
likely to be incorporated and have an effect on state processes 
than has thus far been the case on a national scale. ^ 
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College Rankings 
and 




Key Considerations 



College rankings have penetrated the public consciousness as one measure of college quality and 
may be being used by policymakers who are looking for more information on which to base deci- 
sions and gauge the effectiveness of postsecondary education. As discussed in the previous section, 
several states formally are using college rankings as one tool to evaluate the relative positioning of 
major research universities, particularly in the international context. Although this limited use may be 
appropriate, precaution is warranted in expanding the use of college rankings in policymaking. From 
a review of literature and the interviews conducted for this brief, several key themes emerged relating 



to the nature of college rankings and their potentii 

Rankings inform pubiic notions of coiiege quaiity. 

Rankings of academic quality are not a new phenomenon, 
although traditionally they were limited to federal and state 
agencies, academic administrators, and higher education 
researchers (Stuart 1995). With the development of the USNWR 
college rankings and the subsequent proliferation of rankings 
from a variety of sources, awareness has spread rapidly among 
the general public. This awareness has led to the democratiza- 
tion of knowledge about colleges and the higher education 
sector, as students and families now have access to data and 
qualitative information that was not publicly tabulated or acces- 
sible in the past (McDonough, Antonio, Walpole, & Perez 1 998) . 
The popularity and durability of college rankings are the result 
of several conditions. 

• The increasing mobility of students has created a national 
higher education marketplace, in which institutions and 
students seek information on colleges that is at least loosely 



impact on the policy process. 

comparable. The wide and growing income gap between 
high school graduates and college graduates has raised the 
stakes for gaining admission to the “best” colleges. And it is 
not enough for a person to receive a high-quality education; 
to extract economic value, others must recognize the quality 
of the degree (Ehrenberg 2005; Frank 2001). 

• The use of rankings by postsecondary institutions has contrib- 
uted to their popularity. Despite the shortcomings of many 
ranking systems, institutions are concerned about their posi- 
tion and often use rankings in marketing materials to commu- 
nicate quality and prestige (Ehrenberg 2002). This use of 
rankings by the institutions suggests to the public that rank- 
ings are a reliable source of information in the complex land- 
scape of higher education. 

• Rankings are becoming increasingly important in the interna- 
tional context. They are a critical aspect of discussions of 
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higher education quaiity internationaiiy, because of giobai 
expansion in access to higher education and consumers’ 
desire for information on the quaiity of coiieges around the 
worid (Diii & Soo 2005). 

Data limitations restrict the usefulness of college 
rankings to poiicymakers. 

The compiiation of coiiege rankings is iimited by the avaiiabiiity 
of credibie and comparabie data indicators. The majority of 
ranking systems reiy on nationaiiy comparabie input measures 
(e.g., SAT scores of admitted students, acceptance rates, 
percentage of students in the top 10 percent of their high schooi 
ciass) and contextuai factors (e.g., reputation, aiumni giving rates, 
percentage of professors with a terminai degree in their fieid). 
These measures are reported by institutions, drawn from nationai 
datasets such as integrated Postsecondary Education Data 
System (iPEDS), or coiiected specificaiiy by a rankings pubiisher. 

Poiicymakers who focus on coiiege rankings run the risk of 
overiooking more nuanced data that may be avaiiabie from 
other sources. Persistence and program compietion data are 
exampies of this phenomenon. Many states have constructed 
student unit record data systems to assess the performance of 
their pubiic institutions and generate internai assessments of 
institutionai performance (Eweii & Boeke 2006). in some cases, 
poiicymakers may find that the data compiied by their own 
jurisdiction through these data systems paint a more compiete 
picture of the successes and chaiienges of higher education in 
their state than that provided by coiiege rankings. 



The structure of rankings limits the transference of 
information relevant to policy. 

Ordinai rankings are inherentiy misieading. Rankings are 
presented as accurate and vaiid refiections of the differences 
among the coiieges they rank when they actuaiiy ignore some 
information by design (Stake 2006). Uniike summary scores, 
rankings impiy a consistent magnitude of difference, making 
institutions appear more or iess different in overaii quaiity than 
they actuaiiy are. True differences among cioseiy ranked insti- 
tutions may be minimai or quite iarge, but aii rankings are 
presented as having the same magnitude of difference. 

For exampie, according to the most recent iteration of USNWR’s 
ranking of Historicaiiy Biack Coiieges and Universities, the 
difference in measured quaiity between Speiman (#1) and 
Morehouse (#3) is greater than the expressed difference 
between schooi number 11 and schooi number 33. in other 
words, according to the USNWR scoring metrics, any two 
schoois between 11th and 33rd piace in the ranking are iess 
different in quaiity than the schoois ranked first and third 
(USNWR 2009). Thus, 22 piaces near the middie third of the 
rankings represent about the same difference as two piaces at 
the top. This dynamic suggests that poiicymakers shouid be 
carefui about how they use rankings in their deiiberations of 
differences of educationai quaiity; significant shifts in ranking 
position may reflect minimally if at all on changes in institutional 
practice or quality. 

Beyond distortions of differences in quality among institutions, 
rankings suggest that quality is a finite resource by benchmarking 



INSTITUTE FOR HIGHER EDUCATION POLICY 



15 



to top-performing institutions. For example, in the USA/l/l/fl rankings 
institutions are benchmarked to best performers in each cate- 
gory for which data are collected. This approach reports quality 
in terms of relative performance of higher education institutions. 
Presenting information in this way may be appropriate to iden- 
tify how institutions are functioning compared with other institu- 
tions, but it is of limited use in crafting answers to policy relevant 
questions, such as “Are we producing the educational outcomes 
we desire?” and “What improvements have we made over time 
based on policy changes?” 

Question such as these are critical for policymaking in higher 
education. Educational outcomes may not be best judged on a 
relative basis; it is more useful to determine whether outcomes 
are meeting clearly articulated needs and performance targets. 
It is of little help to know that colleges in one jurisdiction are 
doing “better” than those in other jurisdictions without contex- 
tualizing the knowledge in policy-relevant ways. An institution’s 
rise in the rankings might reflect improvement resulting from 
policy changes, or it could just as easily be due to poor perfor- 
mance by former leaders in the rankings. Moreover, position in 
a ranking reveals limited information on the contributions an 
institution may be making in support of policy goals. Policy- 
makers need to consider how methodologies and reporting 
styles can affect the policy relevance of information communi- 
cated by rankings. 



Rankings have the potential to shift institutional 
behaviors in ways that may negativeiy affect pubiic 
poiicy goais. 

College rankings can encourage schools to become more like 
what the rankings measure, by imposing a standardized defini- 
tion of educational quality and creating an incentive to conform 
to that definition (Espeland & Sauder 2007). Additionally, the 
kind of data used to construct rankings incentivizes specific 
institutional behaviors. The inputs in common ranking 
systems — and the indicators selected to measure them — reflect 
implicitly value-laden decisions about how to appropriately 
define educational quality. Thus, when rankings enter govern- 
ment policy deliberations, attention must be paid to ensure 
definitions of educational quality represented by rankings align 
with policy goals; access and equity policy are areas where 
doing so is particularly important. 

Most of the indicators used in the construction of college rank- 
ings have little to do with policy goals relating to access and 
equity; they create uniform notions of educational quality and 
overlook important distinctions in educational preparation, 
personal experiences, and historical treatment of various 
student populations in higher education. Policymakers and the 
public are ill-served by rankings that rely on data indicators that 
by their nature are exclusionary. Policymakers should take note 
of the data inputs used to construct rankings and ensure that 
incentives for an institution to move up the rankings do not run 
counter to public policy goals, especially in the areas of access 
and equity. ^ 
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Conclusions and 
Recommendations 



College rankings are one piece of information available to policymakers as they usher higher education 
policy through the complex process of policymaking. This brief has provided an overview of national 
postsecondary assessment efforts and noted the similarities and differences in these approaches to 
college rankings, presented examples of the inclusion of college rankings in state government assess- 
ment efforts, and highlighted key themes for policymakers to consider in appraising the usefulness of 
college rankings to policymaking. 



Notably, policymakers need to understand the effects rankings 
can have on institutional behaviors, publicly constructed 
notions of educational quality, and equity outcomes. Rankings 
are likely to remain a fixture of the college information market- 
place. Ensuring that policymakers are aware of these potential 
effects and improving the policy relevance of rankings are 
worthy endeavors. In support of this effort the following recom- 
mendations are offered: 

Take precautions to ensure that college rankings are 
only used as part of overall system assessment efforts 
and not as a stand-alone evaluation of colleges. 

If policymakers are to use college rankings as part of assess- 
ment systems, they must be careful to do so as part of compre- 
hensive efforts designed to improve institutional performance 
toward public policy goals. Although rankings can serve as one 
metric for assessing institutional progress, policymakers should 
use them as part of a larger feedback loop that ultimately leads 
to desired changes in institutional actions and policy creation to 
support those changes. Using rankings as stand alone measures 
of institutional success seldom serves public policy goals. 

Moreover, using rankings as a stand alone evaluation risks 
complacency among public officials in jurisdictions with institu- 
tions that rank high and creates a temptation to use the rank- 
ings as a bludgeon against poorly ranked institutions. Neither 



approach advances the public agenda of higher education. 
College rankings can be part of the process of postsecondary 
policymaking, but policymakers should carefully consider what 
is being measured, the methods used to create a ranking, and 
how these factors relate to public policy goals. 

Support the collection of data that can be used to 
craft more policy-relevant college rankings, inciuding 
providing funds to higher education institutions to 
wideiy impiement and pubiish the resuits of student 
iearning assessments. 

To be of more than nominal use to policymakers, college rank- 
ings need to rely on indicators that reflect policy priorities and 
capture the kinds of skills that make students competitive and 
productive in a modern technological economy. Focusing on 
student learning outcomes is one way rankings could become 
more relevant to policymakers. 

Because of the limited availability of student learning outcome 
data, ranking systems do not incorporate these data. Although 
certain assessment efforts — the VSA, for example — are 
attempting to more comprehensively capture outcome data, 
the lack of a single measuring device is problematic. Rankings 
rely on consistently reported data, so policymakers should 
financially support efforts to capture educational outcomes in a 
consistent format. The use of outcome measures — particularly 
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when presented in terms of value-added scores — in rankings 
would create an incentive for institutions to focus on improving 
educational practice rather than chasing notions of quality 
through exclusivity. Such a focus would help the higher educa- 
tion sector meet public policy goals in the areas of student 
persistence, completion, and equity. 

Leverage public attention to college rankings to 
shape general notions of college quality and advance 
equity goals. 

Policymakers should leverage the attention paid to college 
rankings to shape public discussions of educational quality to 
advance equity goals, an increasingly important undertaking 
given shifting demographic patterns and the need to better 
serve students of color, adult learners, and low-income popula- 
tions to reach national educational attainment goals. 

Policymakers have an opportunity to shape public discussions 
on the effects of rankings on college quality and equity. Currently, 
the indicators used in the construction of rankings have minimal 
relevance to policy focusing on access and equity; instead, 
they reward exclusivity and prestige (Carey 2006). Such indica- 
tors convey little information about the educational experiences 
of adult students, students of color, and low-income popula- 
tions — groups that postsecondary education must better serve 
to achieve national goals for educational attainment (Lumina 



Foundation for Education 2009). Policymakers should be 
publicly discussing why equity outcomes are important policy 
goals given the nation’s shifting demographic patterns, and 
how notions of college quality can be reframed to support 
those goals. 

College rankings have the potential to create incentives for 
institutions to support the educational development of tradi- 
tionally underserved populations. When quality is defined by 
indicators that reward exclusivity, institutional innovation to 
educate underserved student populations more effectively is 
undervalued. Policymakers have an opportunity to shift conver- 
sations about college quality by supporting elements of ranking 
systems that are relevant and useful to public policy in this area. 

Such a deliberative approach is a departure from current prac- 
tice. Focused efforts will be required to move beyond the 
simple inclusion of college rankings in assessments toward a 
public discussion of how rankings affect notions of college 
quality and educational equity, and how they can be most 
useful in policymaking. If policymakers are willing to engage in 
such a process, they can enhance and shift definitions of college 
quality and shape public understanding in ways that will advance 
policy agendas. ^ 
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Overview of College 
Ranking 




A brief overview of several college ranking systems shows the kinds of ranking information currently 
available to stakeholders of American higher education. The five ranking systems presented here 
are “America’s Best Colleges” {U.S. News and World Report) ] “1 00 Best Values in Public and Private 
Colleges” {Kiplinger’s)] “Top American Research Universities” (Center for Measuring University 
Performance); “World University Rankings” (T/mes Higher Education Supplement)] and “Academic 
Ranking of World Universities” (Shanghai Jiao Tong University). These five were selected because 
of their popularity with the public and or the academic community. 



U.S. News and World Report: 

“America’s Best Colleges” 

Since 1983, USNWR has produced undergraduate college 
rankings under the “America’s Best Colleges” banner. This 
ranking has become popular with the public and has captured 
the attention of the higher education community, policymakers, 
academic scholars, and the popular press. Intended to assist 
students and families in their college choice decisions, the 
rankings provide numeric rankings based on quantitative 
measures that cover seven broad categories: peer assess- 
ment; graduation and retention rate; faculty resources; student 
selectivity; financial resources; alumni giving; and, for select 
institutions, graduation rate performance. Scores for each 
measure are weighted by the magazine’s editors and are 
analyzed to arrive at a final overall score for each institution. 
FIGURE 1 shows the weights assigned to each measurement 
category for the most recently released rankings. The overall 
scores are placed in ordinal rank to create the various ranking 
lists. Undergraduate institutions are ranked in peer categories: 



national universities, liberal arts colleges, master’s-level univer- 
sities, and baccalaureate colleges. The latter two categories 
are further subdivided into four geographic regions (U.S. News 
and World Report 2009) . 

Kiplinger’s Magazine: 

“100 Best Values in Public and Private Colleges” 

Kiplinger’s ranks select four-year undergraduate institutions in 
the United States according to the editors’ notions of value, 
producing separate rankings for public and private institutions. 
The rankings are based on a combination of affordability and 
perceived academic quality. Key academic indicators used in 
the rankings include admission rates, first-year student reten- 
tion rates, student-to-faculty ratios, percentage of first-year 
class scoring 600 or higher on the verbal and math compo- 
nents of the SAT or scoring 24 or higher on the ACT, and four- 
and six-year graduation rates. Financial indicators include total 
cost of attendance, (tuition, mandatory fees, room and board, 
books) average cost for students with need after subtracting for 
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FIGURE 1 



Undergraduate Ranking Criteria and Weights 
in USNWR's “America’s Best Colleges” 



FIGURE 2 



Times Higher Education Suppiement 
World University Rankings Weights 



for national universities and liberal arts colleges. 




20% 



category of indicator 


INDICATOR 


EXPLANATION 


WEIGHTING 




Academic 
Peer Review 


Composite score drawn from peer review 
survey divided into five subject areas. 


40% 


Peer Assessment 


Empioyer 

Review 


Score based on received 
responses to employer survey. 


10% 


Retention 
Faculty Resources 


Facuity: 
Student Ratio 


Score based on facuity: student ratio. 


20% 


Student Selectivity 
Financiai Resources 


Citations per 
Facuity 


Score based on research performance fac- 
tored against the size of the research body. 


20% 


Graduation Rate Performance 
Alumni Giving 


internationai 

Facuity 


Score based on proportion 
of international facuity. 


5% 




internationai 

Students 


Score based on proportion 
of international students. 


5% 



SOURCE: WWW.USNEWS.COM/ARTICLES/EDUCATION/BEST-COLLEGES/2009/08/19/ 
METHODOLOGY-UNDERGRADUATE-RANKING-CRITERIA-AND-WEIGHTS.HTML 



SOURCE: WWW.TOPUNIVERSITIES.COM/WORLDUNIVERSITYRANKINGS/METHODOLOGY/ 
SIMPLE_OVERVIEW 



grants, average cost for a student without need after subtracting 
non-need-based grants, average percentage of need met by 
aid, and average debt accumulated prior to graduation. Rank- 
ings are compiled to reflect costs and quality for both in-state 
and out-of-state students. Kiplinger’s places more weight (2:1) 
on academic indicators and breaks ties in the rankings using 
academic quality scores and average debt at graduation. 
Kiplinger’s rankings are noteworthy for their inclusion of at least 
partial data on college cost indicators, which are notably absent 
from most college ranking systems (Kiplinger 2009). 

Times Higher Education Suppiement: 

“World University Rankings” 

The “World University Rankings” are an international comparison 
of institutions. First released in 2005, the rankings rely on four 
areas of evaluation including research quality, teaching quality, 
graduate employability, and international outlook. To represent 
each of these areas, the publisher assigns weighted indicators 
based on its opinion of the importance of the criteria and the 



effectiveness of the indicator to evaluate the measure (Times 
Higher Education Supplement 2008). FIGURE 2 illustrates the 
weighting of the indicators used to determine the ranking. 

Final institutional scores are compiled by multiplying each indi- 
cator score by its weighting factor, adding the resulting figures, 
and scaling results to the top-performing institution. To be 
included in the rankings, an institution must apply and be 
accepted. The most recent list ranks institutions from 1 through 
400. Institutions ranked below 400 are listed in alphabetical 
order (Times Higher Education Supplement 2008). 

Shanghai Jiao Tong University: 

“Academic Ranking of World Universities” 

Shanghai Jiao Tong University’s “Academic Ranking of World 
Universities” compares information across institutions according 
to several indicators of academic or research performance, 
including: alumni and staff winning Nobel Prizes and Fields 
Medals, highly cited researchers, articles published in Nature 
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FIGURE 3 



Academic Rankings of World Universities: 



Ranking Weights 

CRITERIA INDICATOR 

Quality of Alumni winning Nobel Prizes and Fields 

Education Medals 


WEIGHTINGS 

10% 


Quality of 


Staff winning Nobel Prizes and Fields Medals 


20% 


Faculty 


Flighly cited researchers in 21 broad subject 
categories 


20% 


Research 

Qutput 


Articles published in Nature and Science* 

Articles indexed in Science Citation Index- 
expanded, and Social Science Citation Index 


20% 

20% 


Per Capita 
Performance 


Per capita academic performance 


10% 



* FOR INSTITUTIONS THAT SPECIALIZE IN THE HUMANITIES AND SOCIAL SCIENCES— SUCH AS THE 
LONDON SCHOOL OF ECONOMICS— THIS INDICATOR IS NOT CONSIDERED, AND ITS WEIGHT IS 
SHIFTED TO OTHER INDICATORS. 

SOURCE: WWW.ARWU.ORG/RANK2008/ARWU2008METHODOLOGY(EN).HTM#M1 . 



and Science, articles indexed in major citation indices, and the 
per capita academic performance of an institution. For each 
indicator, the highest scoring institution is assigned a score of 
100, and other institutions are calcuiated as a percentage 
of the top score, figure 3 shows the weights assigned to each 
indicator (Center for World-Class Universities 2008). 

These five ranking systems are exampies of the types of infor- 
mation available in the marketplace to policymakers and the 
generai public. Each college ranking system has its own aims 
and purposes, is intended for specific audiences, and reflects 
its own perspective of what constitutes college quality and how 
that quality can be appropriately measured. No one ranking 
system can be considered a “best practice”; rather, each provides 
a range of information that policymakers and other stakeholders 
can consider as they develop notions of college quality. ^ 
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